Image-based localization for addresses

ABSTRACT

The present localization system determines a precise location for an address based on images. Some implementations identify the precise location using one image. Other implementations identify the precise location using multiple images. While still other implementations identify the precise location by interpolation, which uses precise locations of addresses nearby that were previously localized using one or more images. The images used in determining the precise locations have a visual feature associated with the address appearing in the image.

BACKGROUND

Geocoding assigns geographic codes or coordinates (e.g., longitude andlatitude) to map or locate features, such as street addresses. Once themapped features are assigned geographic coordinates, the features may bemapped and entered into a geographic information system (GIS). Mappingapplications may utilize the GIS to obtain the geographic coordinatesand display a symbol (e.g., star, push-pin) on a computer-generated mapthat corresponds to a location for the map feature.

The placement of the symbol on the computer-generated map is only asaccurate as the geographic codes or coordinates assigned by a geocodingtechnique (i.e., localization technique). Current localizationtechniques employ address interpolation to determine coordinates forstreet addresses within a street segment. Address interpolation assumesthat addresses along the street segment are evenly distributed. Whenaddresses are not evenly distributed, incorrect coordinates are assignedto the addresses.

Current localization techniques that utilize address interpolation alsofail in other situations. One situation where current localizationtechniques fail occurs when an address is located within a largerbuilding, such as a mall or a multi-stored complex. Because thelocalization techniques produce a point for each address, an address isnot accurately handled if the address is distributed along the length ofa segment, such as an address for a shopping mall. For example, thesymbol for a store at one end of the shopping mall may be located on thecomputer-generated map at the opposite end of the shopping mall. Anothersituation where current localization techniques fail occurs whenaddresses are on multi-level streets.

As one can see, current localization techniques that utilize addressinterpolation are neither ideal nor precise.

SUMMARY

Described herein are various technologies and techniques for localizingaddresses using images. Some implementations identify a location for anaddress based on one image. Other implementations identify the locationbased on multiple images. While still other implementations identify thelocation for the address by interpolation, which uses addresses nearbythat were previously localized using one or more images.

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used to limit the scope of the claimed subject matter.

BRIEF DESCRIPTION OF THE DRAWINGS

Many of the attendant advantages of the present localization system andtechnique will become more readily appreciated as the same becomesbetter understood with reference to the following detailed description.A brief description of each drawing is described below.

FIG. 1 illustrates one example technique for obtaining images used indetermining precise locations for addresses.

FIG. 2 illustrates one example configuration of localization componentssuitable for implementing a localization system that determines preciselocations for addresses.

FIG. 3 is a flow diagram illustrating one example of a process fordetermining a precise location for an address using images.

FIG. 4 is a flow diagram illustrating one example of a process thatutilizing the precise locations determined by the process shown in FIG.3.

FIG. 5 is a functional block diagram of a computing device that mayimplement one or more of the localization components shown in FIG. 2.

DETAILED DESCRIPTION

The following discussion first describes an example technique forobtaining images used in determining precise locations for addresses.Next, the discussion focuses on localization components suitable forimplementing a localization system that determines precise locations foraddresses. The discussion then describes an example process suitable fordetermining precise locations for addresses using images and an exampleprocess that utilizes the precise locations in a mapping application.Lastly, the discussion describes one possible configuration for acomputing device that may implement one or more of the localizationcomponents.

However, before describing the above items, it is important to note thatvarious embodiments are described fully below with reference to theaccompanying drawings, which form a part hereof, and which show specificimplementations for practicing various embodiments. Other embodimentsmay be implemented in many different forms and should not be construedas limited to the embodiments set forth herein; rather, theseembodiments are provided so that this disclosure will be thorough andcomplete. Embodiments may take the form of a hardware implementation, asoftware implementation executable by a computing device, or animplementation combining software and hardware aspects. The followingdetailed description is, therefore, not to be taken in a limiting sense.

In various embodiments, the logical operations may be implemented (1) asa sequence of computer implemented steps running on a computing deviceand/or (2) as interconnected machine modules (i.e., components) withinthe computing device. The implementation is a matter of choice dependenton the performance requirements of the computing device implementing theembodiment. Accordingly, the logical operations making up theembodiments described herein are referred to alternatively asoperations, steps, or modules.

Example Technique for Obtaining Images

FIG. 1 graphically illustrates one example technique for obtainingimages used in determining precise locations of addresses. A city street100 is shown on which several businesses (e.g., barber shop 102, café104, and grocery store 106) are located. Each business has astreet-level view (e.g., street-level view of café 110), which is a viewas seen from an observer who is passing by the business at street-level.An image-capturing device 120 captures an image (e.g., image 122) of thestreet-level view 110. The image 122 includes one or more street-levelfeatures 124, such as facades of buildings, store logos, store signs(e.g., “CAFÉ”), house number signs, and other features that distinguishthe business from other businesses nearby.

In addition, another device 112 obtains positional data 114 associatedwith image capturing device 120. The positional data 114 includesgeographic coordinates related to image capturing device 120 when image122 is captured. The positional data 114 and image 122 are associatedwith each other. The association may be achieved manually by writing thepositional data and the picture number, may be achieved by recording thepositional data and picture number in an electronic device, or may beachieved using other techniques whereby the image and positional dataare associated. Multiple images may be taken of the same business withthe same or different image-capturing device. Typically, each of themultiple images is taken at a slightly different angle and/or positionso that the street-level view is a little different in each image. Eachimage and associated positional data is recorded.

Image capturing device 120 may utilize any well-known image capturingtechnology found in digital camera, digital video camera, and othercapturing devices. For example, the images may be captured by a personwalking on the street who is taking pictures with a digital camera, maybe captured by a camera positioned on or in a vehicle that is drivingdown the city street, or may be captured using other techniques. Thepositional data may be obtained using a global positioning system (GPS)device that is carried by a person, mounted on or in a vehicle, or maybe obtained by any other device configured to obtain positional data.For example, an automobile may be equipped with a GPS device. When apicture is taken, the GPS readings are recorded and associated with thepicture (i.e., captured image). The images and the respective positionaldata may be stored in an image database that is accessible to alocalization system. While FIG. 1 illustrates one example technique forobtaining images and respective positional data, the presentlocalization system may obtain this information in various other wayswithout departing from the scope of the claimed invention.

Components for Implementing a Localization System

FIG. 2 illustrates one example configuration of localization componentssuitable for implementing a localization system. FIGS. 3 and 4illustrate example processing performed by one or more of thesecomponents. In overview, the localization system 200 includes images202, positional data 204, an entity database 206, and a localizationtechnique 208.

Images 202 include a set of captured images that covers a specifiedarea. The images 202 may have been captured as illustrated in FIG. 1 orby another technique. The area covered by the set of captured images maybe a residential neighborhood, a business district, or any other areahaving structures that are located within proximity to each other. Inaddition, the area may be of any size. For example, the area may be aresidential block, a strip mall, a city, a county, a state, or othersized area. In contrast with satellite images that provide a “roof top”view, the set of captured images provide a “street-level” view from auser's perspective as if the user is driving or walking in the specifiedarea. Therefore, the set of captured images typically includes facadesof buildings, store logos, store signs, house number signs, and otherstreet-level features.

The positional data 204 identifies a location. The positional data mayinclude a longitude and latitude reading that identifies the location.The positional data 204 may be obtained using well-known devices, suchas a Global Positioning System (GPS) device. Each image in the set ofcaptured images is associated with positional data that indicates thelocation where the image was captured. Positional data 204 may beassociated with each captured image manually or electronically.

The entity database 206 includes one or more entities (e.g., entity228). Each entity is associated with at least one visual feature (e.g.,visual feature 230), such as a landmark, a logo, a previously knownimage, or other visual information that identifies a specific entity.For example, the visual features 230 in database may be logos obtainedfrom a yellow page book for a city, multi-city area, region, state, orother sized area. Each logo may be associated with a business located atone or more specified addresses identified in the yellow page book. Theentity database 206 also includes an address (e.g., address 232)associated with each entity. The addresses in the entity database 206cover a predetermined area, which may be one of various sizes. Thepredetermined area covered by the addresses may correspond to aspecified geographic area covered by the set of captured images or mayinclude less area than the set of captured images. If the predeterminedarea is larger than the area covered by the set of captured images, thelocalization technique 208 may use interpolation to obtain preciselocations for the addresses associated with visual features that do notappear in any of the images 202. However, as will be explained in moredetail below, the interpolation is based on precise locations calculatedin accordance with the present localization technique.

The localization technique 208 inputs information from the entitydatabase 206, images 202, and positional data. A source for the imagesand the positional data may be the same if the images 202 have alreadybeen associated with their corresponding positional data 204. Thelocalization technique 208 attempts to match a visual feature identifiedin the entity database 206 for one entity to content within one or moreof the captured images. FIG. 3, described later, illustrates an exampleprocess suitable for implementing the localization technique 208.

The localization system may also include a precise location database210. Database 210 stores the outcome (e.g., precise locations) foraddresses processed by the localization technique. In addition, preciselocation database 210 may store one or more captured images associatedwith an address in the entity database 206. Because one or more imagesare associated with an address, the precise location database 210 can bestructured in a manner that facilitates efficient retrieval of imagesrelevant to a requested address. Content of the precise locationdatabase 210 may be used by mapping applications, may be used to updategeographic information systems (GIS) data, or may be used by otherprocesses in which precise locations are desired.

The localization system may also include a retrieval component 212. Theretrieval component accesses the precise location database 210 andobtains a precise location for an address, such as an address entered bya user via a mapping application 214. The mapping application uses theprecise location to determine a placement for a symbol 240 on acomputer-generated map 242 that represents a vicinity around theaddress. In addition, the mapping application may display an image 244associated with the requested address. The displayed image illustrates astreet-level view for the address. This allows a user to become familiarwith the street-level view of the requested address, which mayultimately help the user locate the address when the user travels to therequested address.

Even though FIG. 2 illustrates storing the precise locations in adatabase for retrieval, one will note that the precise locations couldbe determined in real-time in response to a user query. FIG. 2illustrates one possible arrangement of the localization componentssuitable for implementing a localization system. One skilled in the artwill appreciate that any one of the components may perform processingsteps performed by one of the other components. In addition, additionalcomponents may be added to support the described processing steps. Thus,there may be numerous configurations for the localization components.

Example Processes Suitable for Implementing the Localization Components

The following flow diagrams, FIGS. 3 and 4, provide example processesthat may be used to implement the processing performed by some of thelocalization components shown in FIG. 2. The order of operations inthese flow diagrams may be different from described and may includeadditional processing that is not shown. In addition, not all of theprocessing shown in the flow diagrams needs to be performed to implementone embodiment of the localization system.

FIG. 3 is a flow diagram illustrating one example of a localizationprocess 300 for determining precise locations using images. Process 300is suitable for use in the processing performed by the localizationtechnique 208 shown in FIG. 2. Process 300 is performed for one or moreof the entities identified in the entity database. The objective ofprocess 300 is to determine a precise location for an entity. Process300 achieves this objective by using images to determine the preciselocation for an address associated with the entity. Process 300 beginsat block 302 where an entity is obtained. As discussed earlier, theentity is associated with a visual feature and at least one address. Theentity may be obtained from the entity database. In another embodiment,the entity and associated information (e.g., address, distinct feature)may be supplied directly to process 300 without being retrieved from theentity database. In this embodiment, process 300 may be performed inreal-time to determine a precise location in response to the suppliedinformation.

At block 304, one of the addresses associated with the entity isobtained. One will note that an entity may be a chain store located atseveral addresses. In this situation, each store is associated with thesame distinct feature (e.g., logo), but the addresses are different.Process 300 is performed for each address.

At block 306, the address is used to identify a set of captured imagesout of the images. Each image in the set of captured images haspositional data indicating a location near the address. One will notethat by cataloging the images by their associated positional data,images having positional data indicating a location near the address ofthe entity may be easily identified. The set of images captured near theaddress are referred to as identified image(s).

At block 308, the identified images are analyzed to identify whether thevisual feature appears in any of the identified images. An identifiedimage that contains the visual feature is referred to as a matchedimage. A pattern matching technique may be applied to the identifiedimages to determine if the visual feature appears in any of theidentified images. The pattern matching technique may be performedmanually, electronically, or using a combination of both. Manual patternmatching may be performed by visually comparing the visual feature withthe relevant identified images to identify images having the visualfeature. Electronic pattern matching may be performed using anywell-known feature mapping techniques or other technique that canidentify a visual feature within an image.

At decision block 310, the number of matched image(s) is used todetermine the manner in which a precise location for the address iscalculated. If there is one matched image, processing continues at block312. If there is more than one matched image, processing continues atblock 314. If there are no matched images, processing continues at block316.

At block 312, the positional data associated with the one matched image,along with the matched image, is used to determine a precise locationfor the selected entity. A distance from where the matched image wastaken to the actual building associated with the entity may becalculated to obtain the precise location. As discussed earlier, thepositional data provides coordinates of a location where the image wastaken.

At block 314, the positional data associated with each of the matchedimages, along with the matched images, are used to determine a preciselocation for the selected entity. In overview, a ray may be extended ina view direction of one matched image until the ray intersects withanother ray extended in a view direction of another matched image. Twoor more rays may be used. The location at which the rays meet determinesthe precise location. In one embodiment, three or more matched imagesmay be triangulated to obtain the precise location. Triangulation may beperformed using well-known techniques used in surveying, navigation, andother fields.

At block 316, positional data associated with captured images near theaddress of the selected entity are used to interpolate a preciselocation for the address. In contrast with traditional interpolationtechniques, the interpolation technique employed by the presentlocalization system utilizes precise locations instead of performingapproximations based on assumptions. The precise locations calculatedusing process 300 for nearby addresses are used to interpolate a“precise” location for the entity that does not have an image in whichthe associated distinct features is identified. Because preciselocations of nearby addresses are used, the interpolated preciselocation is more accurate than traditional interpolation techniques.Dead reckoning may be used in interpolating the precise location. Deadreckoning is the process of estimating a current position based on apreviously determined position and advancing that position based onknown information. For example, dead reckoning may utilize pixeldisplacements between successive images to interpolate the preciselocation.

At block 318, the precise location may be optionally stored in a preciselocation database. As discussed above, in one embodiment, preciselocations are preprocessed and are stored in the precise locationdatabase. In another embodiment, the precise locations are processed inreal-time without being stored in a database. In another embodiment, acombination of pre-processing and real-time processing may beimplemented. For example, as addresses are requested for mapping,precise locations are calculated and stored in the precise locationdatabase. If the same address is later requested for mapping, theprecise location may be retrieved from the precise location database.

FIG. 4 is a flow diagram illustrating one example of a process forutilizing the precise locations determined with the process shown inFIG. 3. The example process is for a mapping application that generatescomputer-generated maps. One will note that other example processes mayutilize the precise location to implement the corresponding application.In one embodiment, process 400 accesses the precise locations from adatabase. In another embodiment, process 400 accesses the preciselocations in real-time from process 300. Process 400 begins at block402.

At block 402, a requested address is entered. The requested address maybe entered via a mapping application. At block 404, a precise locationassociated with the requested address is obtained. As mentioned above,the precise location may be obtained from a database or may be obtainedin real-time. At block 406, a symbol is placed on a computer-generatedmap based on the precise location. At optional block 408, an imageassociated with the requested address may be displayed. The image mayshow a street-level view of an entity associated with the requestedaddress.

One Embodiment of a Device that Implements the Localization Technique

FIG. 5 is a functional block diagram of a computing device 500 that mayimplement one or more of the localization components shown in FIG. 2.The following describes one possible basic configuration for thecomputing device 500. The computing device includes at least aprocessing unit 502 and memory 504. Depending on the exact configurationand type of computing device, memory 504 may be volatile (such as RAM),non-volatile (such as ROM, flash memory, etc.) or some combination ofthe two. System memory 504 typically includes an operating system 520,one or more applications 524, and may include program data (not shown).Memory 504 also includes one or more of the localization components 522.This basic configuration is illustrated in FIG. 5 by dashed line 506.

Additionally, computing device 500 may also have other features andfunctionality. For example, computing device 500 may also includeadditional storage (removable and/or non-removable) including, but notlimited to, magnetic or optical disks or tape. Such additional storageis illustrated in FIG. 5 by removable storage 508 and non-removablestorage 510. Computer-readable storage media includes volatile andnonvolatile, removable and non-removable media implemented in any methodor technology for storage of information such as computer readableinstructions, data structures, program modules or other data. Memory504, removable storage 508, and non-removable storage 510 are allexamples of computer-readable storage media. Computer-readable storagemedia includes, but is not limited to, RAM, ROM, EEPROM, flash memory orother memory technology, CD-ROM, digital versatile disks (DVD) or otheroptical storage, magnetic cassettes, magnetic tape, magnetic diskstorage or other magnetic storage devices, or any other medium which canbe used to store the desired information and which can accessed bycomputing device 500. Any such computer storage media may be part ofcomputing device 500.

Computing device 500 may also include one or more communicationconnections 516 that allow the computing device 500 to communicate withone or more computers and/or applications 518. As discussed above,communication connections may be utilized to offload processing from thecomputing device to one or more remote computers. Device 500 may alsohave input device(s) 512 such as keyboard, mouse, pen, voice inputdevice, touch input device, etc. Output device(s) 512 such as a speaker,printer, monitor, and other types of digital display devices may also beincluded. These devices are well known in the art and need not bediscussed at length here.

The processes described above may be implemented usingcomputer-executable instructions in software or firmware, but may alsobe implemented in other ways, such as with programmable logic,electronic circuitry, or the like. In some alternative embodiments,certain of the operations may even be performed with limited humanintervention. Moreover, the process is not to be interpreted asexclusive of other embodiments, but rather is provided as illustrativeonly.

It is envisioned that the present localization system may be utilized inseveral environments. For example, the localization system may beprovided as a service to businesses. The localization system may sendimages that were captured near a business to the business. The businesscould then manually review the images and identify the images in whichthe business appears. The localization system may then utilize thoseidentified images to determine a precise location for the business. Inthis example, the pattern matching step is performed manually by thebusiness. This and other implementations are envisioned.

Although the subject matter has been described in language specific tostructural features and/or methodological acts, it is to be understoodthat the subject matter defined in the appended claims is notnecessarily limited to the specific features or acts described above.Rather, the specific features and acts described above are disclosed asexample forms of implementing the claims.

1. A computer-readable media having computer-executable instructionsthat when executed, perform a method comprising: obtaining an entityfrom a first database, the entity being associated with an address and avisual feature; locating a set of images that were captured in proximityof the address, each image being associated with respective positionaldata; identifying whether the visual feature appears in any of the setof images; and determining a precise location for the entity based on anumber of images in which the visual feature appears.
 2. Thecomputer-readable media recited in claim 1, wherein the number equalsone and determining the precise location for the address is based on thepositional data associated with the one matched image.
 3. Thecomputer-readable media recited in claim 1, wherein the number equalsmore than one and determining the precise location for the addressincludes triangulation using at least two of the images in which thevisual feature appeared to identify the precise location.
 4. Thecomputer-readable media recited in claim 1, wherein the number equalszero and determining the precise location for the address includesinterpolation using at least one previously calculated precise locationfor another entity with a nearby address.
 5. The computer-readable mediarecited in claim 1, wherein locating the set of images includessearching an image database in which the set of images are organized byassociated positional data.
 6. The computer-readable media recited inclaim 1, further comprising storing the precise location in a seconddatabase.
 7. The computer-readable media recited in claim 6, furthercomprising storing a matched image in the second database in associationwith the precise location for the address, the matched image being oneof the images in which the visual feature appears.
 8. Thecomputer-readable media recited in claim 7, further comprisingdisplaying a symbol on a computer-generated map, the symbol representingthe precise location for the address.
 9. The computer-readable mediarecited in claim 8, further comprising displaying the matched image inconjunction with the computer-generated map.
 10. The computer-readablemedia recited in claim 1, wherein the first database comprises a yellowpage directory.
 11. The computer-readable media recited in claim 1,wherein the visual feature comprises a logo.
 12. The computer-readablemedia recited in claim 1, wherein the visual feature comprises a sign.13. The computer-readable media recited in claim 1, wherein the visualfeature comprises a house number plate.
 14. A computer-implementedmapping application comprising: obtaining a precise location for anaddress, the precise location being derived from at least one imagetaken in proximity of the address, each image being associated withpositional data that is used in determining the precise location; andplacing a symbol on a computer-generate map based on the preciselocation for the address.
 15. The computer-implemented mappingapplication recited in claim 14, wherein at least one of the imagesdisplays a visual feature associated with the address.
 16. Thecomputer-implemented mapping application recited in claim 14, whereinthe precise location for the address is retrieved from a database. 17.The computer-implemented mapping application recited in claim 14,wherein the precise location for the address is determined in responseto a request to map the address.
 18. A computer-implemented methodcomprising: identifying a visual feature within an image, the imagebeing associated with positional data that identifies a location wherethe image was captured; and determining a precise location for an entityassociated with the visual feature based on the positional data.
 19. Thecomputer-implemented method recited in claim 18, wherein determining theprecise location comprises calculating a distance from the locationwhere the image was captured to the entity and adding the distance tothe positional data.
 20. The computer-implemented method recited inclaim 18, wherein the positional data comprises a geographic coordinatesobtained via a global positioning device.